{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A brief introduction to Theano\n",
    "===============\n",
    "Theano is a flexible platform for mathematical calculation. It has many amazing features, such as automatic differentiate, paralleling on GPU, friendly to Numpy users. \n",
    "\n",
    "For installation\n",
    "-----------------\n",
    "```\n",
    "sudo pip install theano\n",
    "```\n",
    "If you want a compatibility for GPU, check the page http://deeplearning.net/software/theano/install.html or make sure the NVIDIA driver and CUDA installed successfully.\n",
    "\n",
    "\n",
    "Background \n",
    "----------\n",
    "When you are using the theano, the process you need to do is building an equation/expression/computation graph. I suggest write down the pipeline and equation on paper first, and then just type them in a theano-like style. \n",
    "Theano contains almost all the Numpy functions and some Scipy functions. If you were already familiar with Numpy, everything is going easy.\n",
    "In most cases, theano ask a strict shape maintenance, for example, you can do ```A*B``` with shapes```(1,) (1,)```, but not ```(1,1,1) (1,1)```. A special case is about broadcasting, which is a fancy feature in both theano and numpy.\n",
    "\n",
    "Getting Start\n",
    "-------------\n",
    "Using theano is just like writing an equation, you need variables, constants and operations. \n",
    "Take the case of linear regression as a simple case:\n",
    "The equations:\n",
    "$$X=\\{x_0, x_1, x_2, ..., x_N\\} x_i\\in R $$\n",
    "$$Y=\\{y_0, y_1, y_2, ..., y_N\\} y_i\\in R $$\n",
    "$$W\\in R, b\\in R $$\n",
    "$$\\hat y = Wx+b $$\n",
    "$$Loss = \\frac{1}{2}\\sum(\\hat Y-Y)^2 $$\n",
    "The code:\n",
    "```python\n",
    "X=T.vector()\n",
    "Y=T.vector()\n",
    "W=theano.shared(NP.array(2.0)) #init value 2.0\n",
    "b=theano.shared(NP.array(0.0)) #init value 0.0\n",
    "y_hat = W*x+b #if you are parsing matrix, use T.dot\n",
    "Loss = 0.5*T.sum((y_hat-Y)**2)\n",
    "```\n",
    "BTW, \n",
    "```python \n",
    "import theano\n",
    "import theano.tensor as T\n",
    "import theano.tensor.nnet as NN\n",
    "```\n",
    "\n",
    "There are two types of variables, TensorVariables likes ```T.scalar, T.vector, T.matrix, T.tensor3```, and SharedVariables ```theano.shared(numpy.ndarray)```, the first one usually need an input from the outside and cannot be updated, and the second one takes the ndarray as the initial values and can be updated. Generally speaking, the TensoVariables are inputs and SharedVariables are parameters.\n",
    "\n",
    "After building the equations, we want a runnable function, the theano.function satisfies this. \n",
    "```\n",
    "f = theano.function(inputs=[X, Y], outpus=[Loss, y_hat])\n",
    "```\n",
    "And now, you can feed the data into f and get the results\n",
    "```\n",
    "loss, y_hat = f(data_X, data_Y)\n",
    "```\n",
    "\n",
    "Gradients and Updates\n",
    "----------------------\n",
    "Sometimes, we need to learn some parameters by gradient descent method. In this case, we should use theano.grad and set the updates option in theano.function.\n",
    "```python\n",
    "#calculate the gradient \n",
    "grads = theano.grad(loss, params)\n",
    "\n",
    "#gradient descent \n",
    "param_update = []\n",
    "for p,g in zip(params, grads):\n",
    "    param_update.append((p, p-lr*g)) # apply learning rate\n",
    "    \n",
    "train_func = theano.function([x, y], [loss, y_hat], updates=param_update, allow_input_downcast=True)\n",
    "```\n",
    "theano.grad applies two inputs, the cost(must be a scalar) and the variables. And it returns a list has the same length with the number of variables. Each gradient is also a theano expression, it's no different betwen constructing a gradient by theano.grad or writing an equation by yourself.\n",
    "And the \"updates\" is a list of pairs or an ordered dictionary, each update is a pair following the format(parameter, new_value).\n",
    "\n",
    "Tips\n",
    "----\n",
    "Theano is open-sourced and widely used, you can find the answers to many questions on Theano official website, Stackoverflow, Github issues, Google Groups.   \n",
    "Extending the equations to programs, theano not only provides the way to write an equation, but also a way for programming.   \n",
    "The most useful functions are ```T.switch and theano.scan ```. It provides the function similar to If-Else and For loop.  \n",
    "If you want to speed up your program, the theano profile can help you a lot.  \n",
    "If you want to run on GPU, just add the flag to environment  \n",
    "```\n",
    " THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python xxx.py\n",
    "```\n",
    "If you need the debug information, try\n",
    "```\n",
    " THEANO_FLAGS=mode=DEBUG_MODE \n",
    " THEANO_FLAGS=mode=FAST_COMPILE\n",
    "```\n",
    "This introduction is not enough to deep learning and machine learning, if you want to begin your own deep learning experiment, please read more tutorial about machine learning(you need to know how to split dataset, how to measure the performance, how to design the network ....) and read some examples(you can find some in my own toolbox tutorial https://github.com/QipengGuo/TheanoWrapper)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
